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DETAILED ACTION 



Drawings 

1 . The drawings are objected to as failing to comply with 37 CFR 1 .84(p)(5) 
because they do not include the following reference sign(s) mentioned in the 
description: 110 and 114 (both mentioned on page 9, line 6). A proposed drawing 
correction or corrected drawings are required in reply to the Office action to avoid 
abandonment of the application. The objection to the drawings will not be held in 
abeyance. 



Claim Rejections - 35 USC §112 

The following is a quotation of the second paragraph of 35 U.S.C. 112: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

2. Claims 5 and 28 are rejected under 35 U.S.C. 112, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which 
applicant regards as the invention. 

Referring to claim 5, it is unclear what the variable K represents in the 
dimensions of the matrix and the claimed equation. There is no.definition in the claim 
and the specification does not provide clarification of this matter. In the interest of 
compact prosecution, the examiner assumes that K represents the length of a feature 
vector mentioned in line 3 of the same claim. It is suggested that the claim recite, "...X 
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represents an image matrix that is generated by stacking N feature vectors", each of 
length K, "corresponding to the set of potentially relevant images...." 

Claim 28 recites the limitation "the identifying" in line 15. There is insufficient 
antecedent basis for this limitation. Claim 28 depends directly on claim 27, which 
recites, "identifying of the feature vector, identifying of the number of training 
samples..." (From claim 23) and "identifying how closely the image and the other image 
match each other...." These three instances of 'identifying* something render "the 
identifying" indefinite in claim 28 because it is unclear which instance is the antecedent 
basis for this limitation. In the interest of compact prosecution, the examiner assumes 
that "the identifying" in claim 28 refers to "identifying how closely the image and the 
other image match each other" in claim 27. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1-4, 9-45 and 50 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over U.S. Patent No. 5,893,095 to Jain et al. in view of U.S. Patent No. 
6,347,313 to Ma etal. 

Referring to claim 1 , Jain discloses a system and method for content-based 
retrieval of images. See Figures 1A-7 and the corresponding portions of the 
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specification for this disclosure. Jain teaches the claimed step of "receiving an initial 
image selection" as receiving a target image (248) in Figure 5A and the corresponding' 
portion of the specification, specifically column 1 1 , line 56 et seq. Jain also teaches the 
claimed step of "generating a plurality of query vectors..." as generating a feature vector 
for the target image containing a plurality of primitive vectors for each primitive (low- 
level feature) of the target image used to query the database in Figure 5A and column 
12. Jain teaches the claimed step of "selecting a set of potentially relevant images..." 
as computing distances between primitive vectors of the query feature vector and the 
feature vectors of other images in the database to compare, rank, and select the closest 
matches in Figure 5B and column 12, line 61 to column 13, line 30. 

Jain does not explicitly teach the use of relevance feedback to generate new 
query vectors or feature weights to select a new set of potentially relevant images. 
However, Jain does provide suggestion for doing so in the discussion on query 
refinement or systematic browsing in column 4, line 32 et seq. A user of Jain's system 
could manually generate a new query by choosing an image from the previous result set 
or modifying the parameters (weights) of the previous query. 

Furthermore, Ma discloses a system and method for retrieving objects (images) 
based on relevance feedback. See the Abstract, Summary of the Invention, Figure 7 
and the corresponding portion of the Ma's specification for this disclosure. Ma's system 
generates query feature vectors for a query image as in the first two steps of claim 1 . 
See column 9, line 63 et seq. for this disclosure. Next, Ma's system selects a set of 
potentially relevant images and displays them to the user by mapping the query feature 
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vectors to a feature space and selecting images in clusters nearest those vectors. This 
corresponds to the third step of claim 1 . Ma teaches the fourth claimed step of 
"receiving feedback regarding the relevance of one or more images of the set of 
potentially relevant images" in column 10, line 32 et seq. Finally, Ma's system selects a 
new set of potentially relevant images based on the relevance feedback by adjusting the 
weighting of feature elements within a correlation matrix as in the final two steps of 
claim 1. See column 9, line 21 et seq. and column 10, line 34 et seq. of Ma's 
specification for this disclosure. Ma does not explicitly teach generating new query 
vectors based on the relevance feedback. 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to implement relevance feedback similar to that of Ma's system into 
Jain's system and method such that a user could provide relevance feedback regarding 
the selected group of images and the system would generate new query vectors and 
feature weights (such as in Ma's correlation matrix) to select a new set of potentially 
relevant images. One would have been motivated to do so because of Jain's 
suggestion above and the automated ability of Ma's system to fulfill such a task. 

Referring to claims 2 and 3, Jain's system in light of Ma's disclosure as above 
discloses the system as claimed. Ma's correlation matrix is a full matrix used to 
determine the relevance (distance) between feature vectors in the higher-level feature 
space of clusters. The user's relevance feedback is embedded in the correlation matrix 
to provide weighting of certain features as more relevant than others between clusters 
in the feature space. See column 4 lines 43-63, column 6 line 23 to column 7 line 45, 
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and column 9 lines 21-41 of Ma's specification for this disclosure. Ma's correlation 
matrix is always a full matrix because it represents every cluster within the feature 
space of the database of images regardless of how many images were returned with 
relevance feedback. 

Therefore, Ma does not disclose the use of a diagonal matrix if the number of 
images returned for relevance feedback is less than the number of feature elements in 
the feature vector being compared. In making the combination of Jain in view of Ma as 
above, it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to choose between a full matrix (such as Ma's correlation matrix) 
and a diagonal matrix of simple weights, using the full matrix if the number of relevance 
feedback images was greater than the number of feature elements being compared, 
and using the diagonal matrix otherwise, to determine the distance between feature 
vectors based on the users preferences for weight of each feature. One would have 
been motivated to do so because it would save computational time and storage space 
in maintaining a much smaller matrix than Ma's correlation matrix, depending on the 
number of images returned with relevance feedback. 

Referring to claim 4, Jain's system and method in light of Ma's disclosure as 
discussed above with regard to claims 1-3 discloses the system as claimed. Ma's 
system transforms each feature vector into a higher-level feature space of clusters. The 
correlation matrix is then used to weight the feature elements of the comparing feature 
vectors. These steps correspond to the first half of claim 4. Again, see the sections of 
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Ma's specification mentioned above for this disclosure. The discussion above with 
regard to claims 2 and 3 discloses the second half of claim 4. 

Referring to claim 9, Jain's system and method in light of Ma's disclosure as 
discussed above with regard to claim 1 discloses the system as claimed. The relevance 
feedback of Ma's system is feedback from a user. Furthermore, the suggestion 
provided by Jain's specification mentioned above is also query refinement by a user. 
Referring to claim 10, Jain's system and method in light of Ma's disclosure as discussed 
above with regard to claim 1 discloses the system as claimed. The low-level features 
listed in the claims are all common features used in computer vision to quantify the 
content of an image. The primitives (low-level features) considered in Jain's system are 
local color, global color, structure, and texture. See column 12, line 10 et seq. Also see 
column 3, line 15 et seq. for a listing of other common features extracted in content- 
based retrieval systems. It would have been obvious to one of ordinary skill in the art at 
the time the invention was made to consider a color moments feature, a wavelet based 
texture feature, a water-fill edge feature, and any other common feature used in 
computer vision for content-based image representation as the comparative low-level 
features of the combined system above. One would have been motivated to do so 
because it would be ideal to represent and compare as many features of images as 
possible when using content-based image retrieval as such. 

Referring to claims 11, 12 and 14-18, Jain's system and method in light of Ma's 
disclosure as discussed above with regard to claims 1-3 discloses the method as 
claimed. See the discussion regarding claims 1-3 and 9 for the details of this 
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disclosure. Referring to claim 13, Jain's system and method in light of Ma's disclosure 
as discussed above with regard to claims 1-3 discloses the method as claimed. The 
feature elements (primitives) compared for any given weighting within Jain's system are 
always elements of the same feature or primitive type. See column 12, line 61 et seq. 
of Jain's specification for this disclosure. Referring to claims 19-22, Jain's system and 
method in light of Ma's disclosure as discussed above with regard to claims 1-3 
discloses the system as claimed. See the discussion regarding claims 1-3 and 9 for the 
details of this disclosure. Referring to claims 23 and 24, Jain's system and method in 
light of Ma's disclosure as discussed above with regard to claims 1-4 discloses the 
method as claimed. See the discussion regarding claims 1-4 for the details of this 
disclosure. 

Referring to claim 25, Jain's system and method in light of Ma's disclosure as 
discussed above with regard to claims 1-4 discloses the method as claimed. Although 
the claimed equation is not explicitly disclosed, the computational elements and the 
functionality of the equation are present in the references. See Figures 11 and 12 and 
the corresponding portions of Jain's specification beginning in column 22, line 43. 
Jain's score (Sf) corresponds to the claimed distance (g). The two feature vectors being 
compared (FV1 and FV2) by Jain's system correspond to the query vector and the 
feature vector as claimed. Jain's system calculates the score (s f ) by calculating the 
distance between each primitive (Sj) and then applying a weight (Wj) to each primitive 
score (Sj). Ma's correlation matrix, discussed in detail above, corresponds to the 
mapping matrix (P) as claimed. In combining the teachings of Jain and Ma as above, it 



Application/Control Number: 09/660,536 Page 9 

Art Unit: 2171 

would have been obvious to one of ordinary skill in the art at the time the invention was 
made to use matrix multiplication (with a weighting matrix) and vector subtraction to 
calculate the score (Sf), instead of applying individual weighting elements to individual 
primitive scores, and integrating Ma's correlation matrix into the multiplication as the 
relevance feedback. One would have been motivated to do so because this would 
reduce computation effort while maintaining the same result as Jain's original equations. 

Referring to claim 26, Jain's system and method in light of Ma's disclosure as 
discussed above with regard to claims 1-4 discloses the method as claimed. Although 
the claimed equation is not explicitly disclosed, the computational elements and the 
functionality of the equation are present in the references. See Figures 1 1 and 12 and 
the corresponding portions of Jain's specification beginning in column 22, line 43. 
Jain's score (Sf) corresponds to the claimed distance (g). The two feature vectors being 
compared (FV1 and FV2) by Jain's system correspond to the query vector and the 
feature vector as claimed. Jain's system calculates the score (Sf) by calculating the 
distance between each primitive (Sj) and then applying a weight (Wj) to each primitive 
score (Sj). This computation (the two equations in column 23 of Jain's specification) is 
mathematically equivalent to the claimed equation. In combining the teachings of Jain 
and Ma as above, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to use matrix multiplication (with a weighting matrix) and 
vector subtraction to calculate the score (Sf), instead of applying individual weighting 
elements to individual primitive scores. One would have been motivated to do so 
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because this would reduce computation effort while maintaining the same result as 
Jain's original equations. 

Referring to claims 27 and 28, Jain's system and method in light of Ma's 
disclosure as discussed above with regard to claims 1-4 discloses the method as 
claimed. See the discussion above regarding claims 1-4 for the details of this 
disclosure. Refer specifically to Jain's disclosure of Comparisons beginning in column 
12, line 61 where the 'final score 1 between two images is computed as a weighted sum 
of the individual scores (distances) for the primitives (features). Referring to claim 29, 
Jain's system and method in light of Ma's disclosure as discussed above with regard to 
claims 1-4 discloses the system as claimed. 

Referring to claim 30, Jain's system and method in light of Ma's disclosure as 
discussed above with regard to claims 1-4 discloses the system as claimed. Jain 
teaches the "query vector generator..." of claim 30 as the Image Analysis Module (122) 
of Figure 1A. See Figures 1A and 5A and the corresponding portions of the 
specification for this disclosure, especially column 12 lines 1-3. Jain also teaches the 
"comparator" of claim 30 as the Image Comparison Module (124) of Figure 1 A. See 
Figures 1A and 5B and the corresponding portions of the specification for this 
disclosure. The functionality steps of the claim are disclosed in the above discussion 
with regard to claims 1-4. It would have been obvious to one of ordinary skill in the art 
at the time the invention was made to implement the comparisons made with 
consideration given to relevance feedback, as above, within Jain's Image Comparison 
Module (124) because that module already makes the normal image comparisons. 
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Referring to claims 31, 32 and 35, Jain's system and method in light of Ma's 
disclosure as discussed above with regard to claims 1-4 discloses the method as 
claimed. See the discussion above regarding claims 1-4, 27 and 28 for the details of 
this disclosure. Referring to claims 33 and 34, Jain's system and method in light of 
Ma's disclosure as discussed above with regard to claims 1-4 discloses the method as 
claimed. Again, see the discussion above regarding claims 1-4 for the details of this 
disclosure. Note that Ma's clusters and correlation matrix represent a mapping of 
feature vectors to a higher-level feature space in which relevance feedback is 
incorporated. See column 9, line 21 et seq. of Ma's specification for this disclosure. 
Referring to claims 36 and 37, Jain's system and method in light of Ma's disclosure as 
discussed above with regard to claims 1-4, 27 and 28 discloses the method as claimed. 
See the discussion above regarding claims 1-4, 27 and 28 for the details of this 
disclosure. Referring to claim 38, Jain's system and method in light of Ma's disclosure 
as discussed above with regard to claims 1-4 discloses the system as claimed. 

Referring to claims 39-44, Jain's system and method in light of Ma's disclosure 
as discussed above with regard to claims 1-4, 27 and 28 discloses the system as 
claimed. See the discussion above regarding claims 1-4, 27 and 28 for the details of 
this disclosure. Referring to claim 45, Jain's system and method in light of Ma's 
disclosure as discussed above with regard to claims 1-4, 27, 28 and 30 discloses the 
system as claimed. See the discussion above regarding these claims for the details of 
this disclosure. Referring to claim 50, Jain's system and method in light of Ma's 
disclosure as discussed above with regard to claim 1 discloses the system as claimed. 
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See Figures 1A and 1B and the corresponding portions of Jain's specification for the 
disclosure of the claimed components. Jain teaches the "client device" of claim 50 as a 
computer or workstation (140a and 140b) in communication with a server in Figure 1B 
and the corresponding portion of the specification. Jain teaches the "collection of a 
plurality of images" of claim 50 as the Database Store of Figure 1 A and the 
corresponding portion of the specification. Finally, Jain teaches the "image server" of 
claim 50 as the Server with Backup Facility (160) of Figure 1B. See the discussion 
above regarding claim 1 for the details of the disclosure of the system's functionality as 
claimed. 

4. Claims 5-8 and 46-49 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over Jain in view of Ma as applied to claims 1 and 2 above, and further in view of the 
article entitled, "Relevance Feedback: A Power Tool for Interactive Content-Based 
Image Retrieval" by Rui et al. 

Referring to claim 5, the Jain reference and the Ma reference do not explicitly 
disclose the means for updating the weights into a weighting matrix. Rui teaches a 
system and method similar to the combination of Jain in view of Ma above. Specifically, 
Rui discloses details of the implementation of relevance feedback into the weights used 
to calculate image distances. See Section V (Weight Updating) beginning in the left 
column of page 648 for this disclosure. Refer specifically to part A of this section for the 
functionality of the equation in claim 5. Although the claimed equation is not explicitly 
disclosed, the underlying functionality discussed in this portion of the Rui reference has 
the same effect as the functionality of the claimed equation. The Rui reference uses a 
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more simplistic implementation in that each piece of relevance feedback for each 
relevant image is individually incorporated into the weighting matrix. 

However, it is common practice in the art to use a weighted covariance matrix to 
accomplish the same task by scalar multiplication of the weight ((det(C)) 1/K ) with the 
covariance matrix (C 1 ). In combining the teachings of Jain and Ma as above, it would 
have been obvious to one of ordinary skill in the art at the time the invention was made 
to use a weighted covariance matrix from the feature vectors of relevant images to 
accomplish the same task disclosed by the Rui reference as above. One would have 
been motivated to do so because this would be a more efficient implementation than the 
Rui reference, but would accomplish the exact same thing. 

Referring to claim 6, the Jain reference and the Ma reference do not explicitly 
disclose the means for updating the weights into a weighting matrix. Rui teaches a 
system and method similar to the combination of Jain in view of Ma above. Specifically, 
Rui discloses details of the implementation of relevance feedback into the weights used 
to calculate image distances. See Section V (Weight Updating) beginning in the left 
column of page 648 for this disclosure. Refer specifically to part B of this section for the 
functionality of the equation in claim 6. Although the claimed equation is not explicitly 
disclosed, the underlying functionality discussed in this portion of the Rui reference has 
the same effect as the functionality of the claimed equation. Specifically, equation (29) 
on page 649 of the Rui references serves the same functional purpose as the claimed 
equation. 
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The Rui reference does not explicitly use a diagonal matrix for the weighting as 
claimed. Instead, it uses a weighting set (W ijk ). However, the weighting set serves the 
same purpose as the claimed weighting matrix (W). Therefore, in combining the 
teachings of Jain and Ma as above, it would have been obvious to one of ordinary skill 
in the art at the time the invention was made to generate a diagonal matrix from Rui's 
equation (29), applied only to the diagonal elements of the matrix, for weighting the 
distances between two images. One would have been motivated to do so because the 
generation of this diagonal matrix would serve the same purpose as Rui's weighting set 
in a more efficient manner. 

Referring to claim 7, the system and method of Jain in view of Ma and Rui as 
discussed above with regard to claims 1, 2 and 5 discloses the system as claimed. The 
references do not explicitly teach generating the new query vectors exactly as claimed. 
However, the Rui reference does teach the normalization of the similarity measures with 
the weighting matrix described above with regard to claim 5. This has the same 
functionality as incorporating the relevance feedback into a new query vector as 
claimed. See sections IV and V of the Rui reference for this disclosure, referring 
specifically to the discussion in section IV, part B. Equations (14) - (17) in light of the 
discussion of the weighting set (Wyk) above have the same effect as the claimed 
equation. 

Instead of incorporating the relevance feedback into new query vectors as 
claimed, the Rui reference incorporates the feedback into a weighting set, and then 
normalizes the query vectors with the weighting set when comparing to other images in 
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the database. It would have been obvious to one of ordinary skill in the art at the time 
the invention was made to incorporate the relevance feedback received into the new 
query vectors by normalizing the feature vectors of the relevant images with their 
relevance weightings. One would have been motivated to do so because the Rui 
reference teaches normalization of the same effect, while the Jain reference provides 
suggestion for doing so in the discussion of query refinement in column 4. 

Referring to claim 8, the system and method of Jain in view of Ma and Rui as 
discussed above with regard to claims 1,2,5 and 7 above discloses the system as 
claimed. See the discussion above regarding these claims for the details of this 
disclosure. The references do not explicitly disclose the method of generating a weight 
for the feature vector distances by the equation as claimed. However, the Rui reference 
does disclose the initialization of the weighting elements and the updating of the 
weighting elements having the same effect as the claimed weight generation. See 
sections III and V of the Rui reference for this disclosure. 

The claimed equation for calculating this weight is a common function in the art. 
The equation is merely the integration of all of the feature vectors for the relevant 
images with their relevance feedback incorporated. In combining the teachings of Jain, 
Ma and Rui as above, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to calculate the weight for a distance between two 
feature vectors of Jain's system using an integral over the feature vectors for which 
relevance feedback was received. One would have been motivated to do so because 
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this would be a more efficient implementation than that of the Rui reference, while still 
maintaining the same functionality. 

Referring to claim 46 and 47, the system and method of Jain in view of Ma and 
Rui as discussed above with regard to claims 1 , 2 and 7 discloses the method as 
claimed. See the discussion above regarding claims 1 , 2 and 7 for the details of this 
disclosure. Referring to claims 48 and 49, the system and method of Jain in view of Ma 
and Rui as discussed above with regard to claims 1, 2 and 8 discloses the method as 
claimed. See the discussion above regarding claims 1 , 2 and 8 for the details of this 
disclosure. 



5. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

U.S. Patent No. 6,173,275 to Caid et al. discloses a system and method for 
content-based image retrieval comparing query vectors to feature vectors of database 
images. 

U.S. Patent No. 5,855,015 to Shoham discloses a system and method for 
content-based image retrieval using relevance feedback. 

U.S. Patent No. 5,933,823 to Cullen et al. discloses a system and method for 
content-based image retrieval using texture analysis. 

U.S. Patent No. 6,408,293 to Aggarwal et al. discloses an object retrieval system 
and method using relevance feedback. 



Conclusion 
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U.S. Patent No. 5,696,964 to Cox et al. discloses a multimedia database retrieval 
system and method based on relevance feedback from a user. 

U.S. Patent No. 5,950,189 to Cohen et al. discloses a system and method for 
object retrieval by calculating the distance between query vectors and feature vectors. 

U.S. Patent No. 6,41 1,953 to Ganapathy et al. discloses a content-based image 
retrieval system and method. 

U.S. Patent No. 5,778,362 to Deerwester discloses a system and method for 
relevance feedback and retrieval of data items. 

The other publications disclose different aspects of content-based image retrieval 
systems and methods using relevance feedback. 

6. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Brian Goddard whose telephone number is 703-305- 
7821 . The examiner can normally be reached on M-F, 9 AM - 5 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Safet Metjahic can be reached on 703-308-1436. The fax phone numbers 
for the organization where this application or proceeding is assigned are 703-746-7239 
for regular communications and 703-746-7238 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is 703-305- 
3900. 
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